fix(ocap-kernel): keep the kernel's account of a vat's life in one piece - #1023

…for endpoints that are gone

Restarting a vat alongside a running run loop cannot be made safe by ordering
alone. The previous approach recorded the vat as mid-flux so a delivery would wait
for the new incarnation, and the record had to be installed *after* waiting out
the crank in flight — install it before, and a crank that is already running
reaches its endpoint lookup, finds the record, and waits for a restart that is
waiting for that crank to end. That ordering left a turn of its own: a crank the
run loop starts between the wait resolving and the record appearing takes the
outgoing handle, and can be mid-delivery when the worker goes down.

So the restart is now the run loop's own work, as a queued `restartVat` item, the
way SwingSet queues `upgrade-vat` for `processUpgradeVat`. In a crank of its own
there is no window to close: the run loop is the only thing that delivers, and it
is here instead, so the vat is idle by construction. `Kernel.restartVat` settles
when the crank has done it, and refuses outright if the run loop is dead, since
nothing would ever carry the request out.

Termination keeps the flux record, because it cannot be queued: `reset` and
`clearStorage` tear vats down on kernels whose run loop has died. Both of its
steps now live inside `#trackFlux`, in the order that does not deadlock, so a
caller does not sequence them and cannot get them wrong — with a test that hangs
if the order is reversed.

Two more, found in review of the previous round:

`#deliverNotify` and `#deliverBringOutYourDead` awaited the endpoint with no
handling for one that has vanished, so a crank landing during a termination took
the rejection into the run loop and killed it. This predates the wait — the lookup
used to throw synchronously in the same case — but the wait is what makes it
routine. All three of notify, reap, and GC-action delivery now go through
`#resolveEndpoint`, which drops the work for an endpoint that is gone for good
(a terminated vat, or a remote) and propagates anything else. The notify resolves
its endpoint before translating, which would otherwise mint c-list entries for an
endpoint with no way to hear about them.

A relaunch that failed marked the vat terminated but left its root pinned:
`stopVat` releases that pin only when it is the one ending the vat, and it had
been told the vat was coming back, while vat cleanup does not touch pins at all.
The pin, and the root's refcount, were held for the life of the kernel. Both
paths now release it through one helper.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The send path caught every endpoint lookup failure and treated it as a splat,
which its own TODO called out: an error that is not "this endpoint is gone"
silently discarded a deliverable message and rejected its result with
ENDPOINT_UNREACHABLE. It is now the last of the four delivery paths to go through
`resolveEndpoint`, so a splat happens where the endpoint will not be back — a
terminated vat, or a remote — and anything else propagates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…nd rollback

Four ways to kill or wedge the kernel, found reviewing this branch.

`rollbackCrank` emptied `maybeFreeKrefs` rather than restoring it. The set is
not per-crank — only `collectGarbage` empties it, at the end of a crank that had
an item — so a candidate created while the run loop was idle, as `terminateVat`
unpinning a root creates one, was owed a collection that any later crank's
rollback silently cancelled. Savepoints now carry the set as it stood when they
were taken. The audit cannot see this one: the counts stay self-consistent at 0.

A restart that could not relaunch its vat threw, and the run loop's catch rolls
back on any throw — undoing the termination records `performVatRestart` had just
written and returning the request to the run queue. Every subsequent process
start dequeued it and failed the same way. It now terminates the vat and reports
through the waiter, so the crank commits and the request is spent. The comment
claiming the throw preserved those records had the causality backwards.

Terminating a vat left a queued restart for it to be carried out against a vat
that no longer existed; `#restartVatWorker` is the one item type that does not
go through `#resolveEndpoint`, so the resulting `VatNotFoundError` propagated.
Restart-then-terminate is reachable from RPC. The waiter is now rejected when
the vat is terminated and the request dropped when the crank reaches it.

`cleanupTerminatedVat` ends by *unmarking* the vat it finished, so work
outliving it — a `bringOutYourDead` scheduled before it died, which nothing
purges from the reap queue — arrived at an endpoint that was neither present nor
terminated, which `#resolveEndpoint` reserves its throw for. It now asks whether
the store has a live record of the vat at all.

Also fixed, from the same review:

- `getImporters` counted only vats, so retiring an object deleted it without
  telling a remote importer, leaving a c-list entry naming nothing — which the
  audit reports as dangling, taking the run loop with it. Adds `getRemoteIds`.
- `#deliverGCAction` computed the live kref set before awaiting the endpoint and
  used it after. A remote re-handshaking in that window clears its c-list
  without waiting for the crank, and `krefsToErefs` throws rather than returning
  short.
- `#endVat` marks the vat terminated in a `finally`. A teardown that threw left
  it unmarked, which is the state above, and falsified `#trackFlux`'s stated
  invariant that waiters can read "gone" as terminated.
- Comments that no longer described the code: `provideVat` waiting on restarts
  (only teardown is recorded), `stopVat` tearing down "only the worker" (it
  releases the root pin, as of this branch), `clearStorage` terminating vats,
  the audit standing in for the disabled `retireExport` assert, and a stale
  `(1, 1)` baseline rationale. `#vatsInFlux` narrows to `Promise<void>`, which
  removes a branch of `provideVat` that could not be reached.

Tests: each fix has a regression test that fails against the code without it.
Closes the two coverage gaps the review named — the splat path charging the run
queue item's own target when routing went through a promise, and `ko6.refCount`
in the control-panel e2e, restored as three per-checkpoint values rather than
dropped as nondeterministic. Full unit suite, kernel-test with auditing on every
crank, and `test:e2e:ci` at 17/17.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Only `deleteVat` removes `vatConfig.<vatId>`, and `cleanupTerminatedVat`
sweeps `${vatId}.`-prefixed keys, which never match it. The writes making up
a vat's death were interleaved with awaits across `VatManager.stopVat`,
`#endVat`'s `finally`, `VatHandle.terminate` and a lambda in `Kernel.ts`, so a
throw part-way left the vat marked terminated with its config alive — which
reads as *active* again as soon as cleanup drops the mark, killing the run
loop over the disagreement and resurrecting the vat on the next process start.

`VatManager.#retireVat` now makes all four writes with no await between them,
modelled on SwingSet's synchronous prelude in `kernel.js` `terminateVat`;
worker teardown follows and is best-effort. `#endVat` and `#abandonVat` go as
duplicates of it, and `VatHandle.terminate` is left with only its own channel
to close.

A vat whose stream fails is retired by the manager, via a new
`onCriticalFailure`, rather than tearing itself down: that left the handle in
the manager and the vat live in the store, so the next delivery went to a
worker that could not answer and, the vat RPC client having no timeout, the
crank never completed while the run loop still reported itself running.

`makeGCAndFinalize` drains the queues before sweeping, since a pending
continuation still holds its closure's objects, so a vat reports its dropped
imports on the `bringOutYourDead` that provoked them rather than a later one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…tial launch terminated

Two conflicts the rebase onto the GC-hardening stack surfaced, both real
disagreements rather than textual ones.

`revertStateBeneathRollback` cleared `maybeFreeKrefs` outright. The set is not
per-crank — only `collectGarbage` empties it — so a candidate added while the
run loop was idle, which `terminateVat` unpinning a root produces, was owed a
collection and lost it to an unrelated crank's rollback. It now restores the
savepoint's snapshot, which discards the abandoned crank's additions and keeps
everything that predates it. The unit test had encoded the old behaviour and is
updated to distinguish the two cases.

`launchVat`'s cleanup relied on `stopVat` reaching `#retireVat` to record the
death, but `stopVat` refuses a vat the kernel has no handle for and the store
does not call active — which is what a partial launch looks like. The mark is
asserted directly again, as it was before this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Recording the vat's death only saves the deliveries that come after it. The one
in flight when the worker died stays parked on an RPC client with no timeout, so
its crank never completes — the same hang `onCriticalFailure` exists to prevent,
one delivery earlier. The worker was left running too, since nothing else would
stop it once the handle was off the books.

Found by Cursor Bugbot on #1023.

Also reverts this branch's additions to the extension control-panel e2e test.
They asserted `ko6.refCount` directly, and which vat owns `ko6` is not stable:
order. The behaviour they checked is covered by the refcount audit, which runs
on every kernel `kernel-test` builds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… exists

`onCriticalFailure` closed over the binding `VatHandle.make` was still to return, so a stream that broke during `#init` threw on the temporal dead zone — after the vat had been retired, and before anything rejected the pending `initVat` that nothing else will ever settle. The handle is now passed to the callback, and `runVat` refuses to put a handle on the books for a vat retired while it was being made.

The teardown also awaited the worker kill before `terminate`, which is what rejects the vat's pending RPCs. A worker slow to die — or one that never does — kept the parked delivery parked, which is the hang this path exists to clear. The two now run alongside each other, `terminate` first.

Found by Cursor Bugbot on #1023.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>